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ABSTRACT 

Reported project objectives were to develop a pupil 
performance profile which describes the effects of specific 
instructional materials and management procedures on student 
performance and to develop a model which predicts pupil performaace 
in a given set of materials under particular conditions. 
Instructional materials (Mathematics, M.W. Sullivan) were employed 
with 50 students (aged 6 to 1 1 years) of whom 24 were classified as 
having learning disabilities and 26 were classified as no3nnal. An 
effort was made to assess only the contribution of the instructional 
materials by keeping contaminating factors such as differences in 
teaching styles or behavior management procedures to a minimum. The 
system was found to make the following information available for 
decision making by teachers: determination of student performance oa 
three major measures by class or type of student, determination of 
performance on each measure by individual student, relationship of 
individuals' performance to that of their reference group (those in 
the same class and/or of the same type) , distribution of performance 
by quartile ranking, and prediction of student performance. For both 
classes, there was found to be an inverse relationship between 
variability in performance within and across instructional materials 
and overall level of performance. (GW) 
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PREFACE 



This project was designed to make several important steps toward the 

full-blown development of a methodology for the evai uation and specif icatjon 
of instructional materials and management procedures, as well as the predic- 
tion of student. performance on instructional materials under various envi- 
ronmental conditions. 

Numerous approaches exist for the evaluation and- specif i.eat ion of 
materials. -However, their success has been- l imi ted by insufficient emphasis 
on systematic analyses of data generated by students working: on Instructional 
materials. Tn; regard to prediction of performance, little has been done to 
enable a teacher to deal with individual students. 

Combining the promising features of existing methods with several novel 
elements, the research focused on the development of a Pupil Performance 
Profile and a predictive model. The Profile and the predictive model uti- 
lized data col lected from several sources: the student whose performance we 

are attempting to assess and predict, and the performance of other -students 
on the same or related materials. The overall goal is to improve learning 
through a comparison of individual student performance to that of other 
students, and the collection of fine-grain data regarding the differential 
effectiveness of various materials and management procedures on students of 
particular characteristics. 







I 




4 



INTRODUCTION 



A strong need exists to be able to specify, in advance, instructional 
materials as well as teaching and management techniques that are most likely 
to change the behavior of any given student. A need of similar magnitude 
exists for the development of more precise procedures for the evaluation of 
both instructional materials and management procedures. 

Lumsdaine (1965) provides a thorough discussion of the evaluation of 
instructional materials, particularly programmed materials. He mentions 
three kinds of standards for evaluation: appropriateness, practicality, 

and effectiveness. The latter, which is most relevant to this research, 
refers to how well the instructional materials accomplished their objec- 
tives; in other words, does it teach what it says it teaches? Several 
indices of effectiveness, such as internal and external , are available. 

The former is revealed through inspection of materials, and involves such 
factors as format, prompting techniques, and kinds of student responses, 
fhe latter deals with information which cannot be secured through observa- 
tion, for example, student performance on the materials. 

As related by Lumsdaine, the two basic purposes of program effective- 
ness assessments are; (1) to provide a diagnostic basis for program 
revision; (2) to provide for the description of performance' characteristics 
of a completed set of materials. Data from the latter indicate to the 
teacher the range of or mean performance to expect on particular instruc- 
tional materials. 

Lumsdaine stresses the importance of identifying the characteristics 
of the student who will be working on instructional materials. This serves 
several purposes: to Identify the entering skills and knowledge of the 

students, so as to permit more precise determination of the amount of prog- 
ress or improvement; to indicate the prerequisite information and skills 
necessary for successful performance on the materials; to enable the deter- 
mination of whether or not the present population of students is typical or 
atypical of the students for whom the materials were designed. 

A major approach used in the specification of instructional materials 
is Individually Prescribed Instruction (IPI). Cox (1966) describes the 
major concerns of IPI as being: (l) the definition of curricula in terms 

of a continuum of specific behavioral objectives, enabling precise neasure- 
ment of pupil performance and progress in various subject-matter areas; 

(2) the provision of a range of instructional materials and techniques to 
meet the individual needs of students. 

IPI seeks to provide conditions which allow students to work at their 
own pace, to learn with a minimum of direct teacher aid, to work only on 
materials which are compatible with their level of competence, and to reach 
certain minimum levels of mastery before proceeding to more difficult 
materi als . 
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The goal of Improved learning is facilitated by frequen*" evaluation of 
student progress. For purposes of evaluation four basic types of diagnostic 
instruments are used: (1) placement, tests ; (.2) pre-tests; (3) curriculum 

embedded tests; (^) post-tests. Teachers and others involved in the learning 
process develop flexible sequences of learning experiences . Curricula are 
expressed in terms of behavioral objectives, and succeeding objectives are, 
when possible, built upon preceding objectives. 

Analysis of placement testing results determines where each student 
should begin working on particular materials. As a result of this testing, 
which diagnoses the student's weaknesses, a prescription Is developed for 
each student. The prescription lists the Instructional materials which 
deal with weaknesses uncovered In the placement testing. By frequent meas- 
urement, through the use of tests at various junctures, and through precise 
specification of materials for particular learning problems, learning is 
faci 1 itated. 

Another recently-developed tool for the evaluation of Instruction is 
Comprehensive Achievement Monitoring (JAM). A major focus of CAM Is the 
discrepancy between expected and actual student performance. As discussed 
by Allen (b?0), tradi tional classroom testing often does not illuminate 
possible differences between expectations and actual performance. Through 
pre- and post-test measurement, CAM, combined with systematic monitoring 
during Instruction, can identify such discrepancies. Once Identified, the 
discrepancies are brought to the attention of the teacher, who can then 
devise more effective instructional tactics. It is the thesis of CAM that, 
unless expected performance levels are precisely defined and specific 
information about performance is available, there will be only slight 
influence on the teacher to change tactics and, hopefully, improve instruc- 
tion. 



Somewhat related to IPI is the Keydex system, devised to aid in the 
specification of materials for students with particular learning problems 
(Adamson and Van Etten, 1970). The user of the system first determines 
which descriptor terms, i.e., the characteristics of particular materials 
indexed in the- Keydex system, are relevant to an academic problem in the 
classroom. At times, no materials may contain the relevant descriptors or 
all of the descriptors. On other occasions many materials may contail all 
relevant descriptors, causing the user to select additional descriptors so 
as to pare down the materials to a manageable number. 

At present, no single approach to evaluation and specification of 
instructional materials is without flaw. Many contain one or more highly 
promising aspects which, when judiciously combined, may prove to be sig- 
nificantly more valuable than any given' aval labia approach. P irhaps the 
major criticism of current practices, whether related to specification or 
evaluation, is that they ijenerally focus on characteristics of the materials 
(Lumsdainc's internal index of program effectiveness), rc^ther than on 
analyses of data generated by students working through the materials. 
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Management procedures are another important aspect of the Instruc" 
tional System. The methods teachers use to increase desirable behaviors 
and decrease undesirable behaviors in the classroom may greatly affect 
student performance. Often, however, the individual teacher is forced to 
devise new methods and procedures on the spot, for lack of validated evi- 
dence concerning the effects of various procedures used in a range of 
s i tuations . 

It seems that a more systematic approach than "try, try again" may 
prove more effective. In general, procedures are changed on a trial-and- 
error basis. If one management technique fails then another is selected, 
almost at random. At the very least the procedure is selected on the basis 
of the experience of the single teacher involved. This is wasteful of time 
and other resources, and may be detrimental to students' intellectual and 
social development, if students often experience failure in the classroom. 

It would be very useful to have information concerning the effects of 
Specific management procedures on specific "types" of students on specific 
instructional materials. Cronbach (I969) has conducted research which 
relates to this aim. He has dealt with the interaction between "learning 
abilities" and instructional treatments, naming this work Aptitude Treat" 
ment Interaction or ATI. The basic premise of ATI is that characteristics 
of learners affect their attainment of educational goals. Two major goals 
of ATI are; ( 1 ) that different instructional methods are needed for dif" 
ferent kinds of students in order to achieve the same instructional objec- 
tive; (2) personality dimensions, as well as aptitude, should be a criterion 
for academic placement. Cronbach observes that the principles governing 
the matching of learner to individualized instruction environment are not 
yet known, and that we have very little information concerning the relation- 
ship of personality variables to instructional procedures. 

We would benefit greatly from objective means of specifying, with a 
particular probability of success, which procedures will work for particular 
"types" of students on particular kinds of materials, before the procedures 
are implemented or materials handed out to the students. Preciseness of 
specification will be based on the accumulated experience and success of 
hundreds of teachers and thousands of students, as contrasted with the 
typical hit-and-miss, "seat of the pants" approach. 



RESEARCH OBJECTIVES 



The project had two basic research objectives. The first objective was 
the initial development of a Pupil Performance Profile, which is a descrip- 
tion of the effects of specific instructional materials and management pro- 
cedures on student performance. This involves specification of how students 
of differing characteristics perform on particular materials under various 
environmental conditions. J,he Pupil Performance Profile includes the 
following information:'^' the *^t,ype" of student involved, i.e., his 
characteristics such as s'cores db/^Jtandardized tests and his grade level; 

(b) the instructional materials involved; (c) the management procedures 
used, if any; (d) student performance, in terms of correct and error rate per 
minute, plus .additional dependent var^iables where relevant. 

.. . A’ 

Efforts to predict student academic performance have generally been 
carried out at the macro-level . For example, many studies have utilized 
personality characteristics or scores on standardized tests to predict grade- 
point average. Results of such predictions have been mixed. Assuming, 
however, that we. could always predict grade-point average from such variables 
as scores on personal 1 ty- inventories, of what value is this to the individual 
teacher dealing with a pai'ticular student working with certain instructional 
materials? 

’*■ 

Therefore, the second objective involved using data from the Pupil 
Performance Profile to begin the development of a model which, enables the 
prediction of pupil performance in a given set of materials under particular 
conditions. The predictive model is intended to serve general purpose func- 
tions, that Is, be usable in any setting, assuming that data is available 
regarding the important variables which are Involved in the prediction 
process. In addition, the model utilizes data generated by the individual 
students whose performance we will be attempting to predict. The following 
outline indicates the relatlorrships between the performance of the individual 
student and the performance of other students: 

Some Possible Sources of Input for the Predictive Model: 

I. Individual Student Measures 

A. Performance on preceding portions of the present set of 
materials, e.g., frames 1-100 of BRL Mathematics . 

B. Performance on related materials, e.g.. Sets and Numbers . 

(Suppes and Suppes, 1968) 

C. Performance on related activities not classifiable as 

materials, e.g., classroom discussion. . , 

, . D. Other relevant data, including, perhaps, students' 

preference for particular ki nds of' materials , particular 
kinds of relnforcers, for working accurately as opposed 
to rapidly, and so on. 
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II. Group (Other Student) Measures 



A. Performance on preceding portions of the present set of 
materials, e.g., frames 1-100 of BRL Mathematics . 

B. Performance on itrimediately upcoming (for the Individual 
student) portions of the present material, e.g., frames 
101-200 of BRL Mathematics . 

C. Performance on related materials, e.g.. Sets and Numbers . 

D. Other relevant data. 

The following chart illustrates the application of the predictive model: 



Performance of 
Individual Student 
on Materials 
(no contingencies 
In effect) 

Frames Correct=2.8/ml n. 
1 - 100 : 

Error=4. 1/ml n . 



Performance of 
Other Students 
on Same Materials 
(w/ contingencies 
in effect) 

Mean correct<=3 .9/ml n . 
Minimum correct=2.6/ml n . 
Maximum correct=5.0/min. 

Mean error=2 .0/mi n . 
Minimum error=0. 7/ml n. 
Maximum error=^.^/mln. 



Performance of 
Other Students 
on Same Materials 
(no contingencies 
in effect) 

Mean cor rect=>3. 5/ml n. 
Minimum correct=2.5/mln. 
Maximum correct=^.7/min. 

Mean error=2.8/mln. 
Minimum error=l .2/mln. 
Maximum error=^.6/min. 



lOi-200: (student has not 
yet begun these 
frames) 



Mean correct=3.0/mln. 
Minimum correct=2.6/min . 
Maximum correct=3.9/mln. 



Mean cor rect=2 .9/ml n . 
Minimum correct=2.7/min. 
Maximum correct=^.l/min. 



Mean error=2 .8/ml n . 
Minimum error=1 .^/mi n. 
Maximum error=4.6/mln. 



Mean error=2 .9/ml n . 
Minimum error=l .5/mln. 
Maximum error=^.8/min. 



Given the above information, how might we structure both the type of 
materials presented to the individual student and the nature of the manage- 
ment procedures in order to Improve his performance? The preceding chart 
supplies several basic pieces of data from which to make such decisions. 
First, the individual student's correct rate on frames 1-100 was lower than 
the other students' mean correct rate on those frames. The individual 
student's error rate on those frames was higher than the other students' 
mean error rate. Second, the mean correct rate of the other students, 
regardless of whether or not a contingency system was in effect, was lower 
for frames 101-200 than it was for frames 1-100. This was accompanied by 
a higher mean error rate for frames 101-200 as compared to frames 1-100. 
Third, the students operating under a contingency system did better, in 
terms of higher correct rate and lower error rate, on frames 1-100 and 101- 
200 than did students not operating under a contingency system. 
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From the preceding, three conclusions may tentatively be drawn: (1) the, 

individual student we are dealing with had considerably more difficulty with 
the Initial (frames l-lOO) portions of the material than did the average 
student. In fact, this student's correct rate v^as very close to the minimum 
rate for the other students; (2) the second 100 frames of the materials are 
more difficult than the first lOO frames, as indicated by both lower correct 
and higher error rates for the other students on the second 100 frames; (3) 
the contingency system produced better performance than did baseline condi- 
tions, for both frames I -100 and 10 1 ”200. 

Armed with this information, we can predict several things about the 
individual student's performance on the immediately upcoming frames (l-lOO): 
First, that we can, all things being equal, expect a decrease in correct and 
an increase in error rate on these frames. Second, we can expect that this 
student will have a lower correct and a higher error rate on these frames 
than did the average of the other students who have worked through the same 
materials. Third, that a contingency system will increase correct and 
decrease error rate on these frames. 

We now have the means for specifying, in advance, the performance of the 
individual student, in addition to knowledge of which management procedure 
will most improve performance. If we also have data regarding the performance 
of students of similar “type" on other materials dealing with the same subject 
matter, we can also predict whether or not the individual student's perform- 
ance will improve or deteriorate if shifted to these other materials. 



Research Methods and Procedures 



Subjects 

Fifty students , of whi ch 2i» were class! fied as having learning disa- 
bilities and 26 classified as normal, were engaged in the project. The 
students with learni ng' disabi 1 ities were taught in a separate classroom from 
the normal children, though all teachers had . comparabi e training and back- 
grounds and used approximately the same teaching techniques. 

The students were from six to eleven years of age. The students clas- 
sified as having learning disabilities were located in a single classroom 
during the time of day that they were involved with the project and returned 
to one of three classrooms when not engaged in the project. The students 
classified as normal exhibited a range of academic and social competence, 
though all fell within the range considered acceptable for their age and 
grade level (seven years, second grade). The learning disabled students 
(ages six to eleven) exhibited a variety of academic and social deficits, 
which led to their placement in special classes. 

Materials 

The instructional materials utilized in the project were Mathe ma tics 
(M.W. Sullivan. Palo Alto, California: Behavioral Research Laboratories , 

lyyo). This is a programmed set of 37 booklets, designed to meet the needs 
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of a wide range of students, from those with no background in mathematics to 
those with fai rly extens i ve experience. Students began on pre-tests provided 
by the publisher. 

Teachers 

The four teachers involved in the project had received at least minimal 
training in precision teaching prior to beginning the project and hence were 
familiar with basic measurement techniques. The normal students were taught 
by a single teacher in a second grade classroom, whereas the learning disabled 
students were taught by a teacher and an aide. 

Behavior Management Procedures 

Essentially the same behavior management procedures were employed by all 
teachers involved in the project, though minor yariances occurred at particular 
times and for particular students when circumstances warranted. The prevailing 
procedure was to utilize only verbal praise, e.g., "good job" or "you are 
correct", contingent upon accurate work. 

Experimental Design 

We attempted to provide comparable conditions, in terms of teachers' 
training, instructional materials utilized, and behavior management proce- 
dures in effect, for the two groups of students involved. The purpose was 
to get as clear a picture as possible of the contributions of the instruc- 
tional materials themselves to student performance, and to keep contaminating 
factors, such as differences in teaching styles cr behavior management pro- 
cedures, to a minimum. 

The project focused upon the following dependent variables: (a) correct 

and error rate on the instructional materials; (b) percentage correct; (c) 
elapsed time per page. In future work, as procedures for gathering and 
analyzing data are further refined, it willbe possible to gather data con- 
cerning such factors as social behavior whi 1e working on the materials, 
retention of information and skills, and students' and teachers' comments 
regarding individual items, pages, and books. In regard to independent 
variables, the focus was upon student type, e.g., learning disabled or normal. 



DATA COLLECTION AMO ANALYSIS 



All students worked for ten minutes per day on the instructional 
materials.. This facilitated comparison of the performances of different 
students by standardizi ng their daily output. Direct readout clocks, 
supplied by the Project, were used by students to record the elapsed time 
each spent working on each page of the materials. A timer was supplied 
each teacher, to determine the end of each of the fixed length periods of 
student work on the materials. 
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Raw data includes the following: page and book identification number; 

date; student by class; number. of movement cycles on the page; number of 
correct movement cycles on the page; number of incorrect movement cycles on 
the page; start time; stop time; frames missed. 

A project aide corrected the students' work and other personnel trans- . 
ferred the raw data to key punch, form. The information was then key-punched. 

A data log (see Appendix A) was constructed on the basis of the dependent 
variables of greatest interest, in addition, to the need for facilitating 
transfer of data both, from the ..instructional materials to the log and from 
the log to computer .cards . .The data log contains the following information:,- 
student identification number, book number, page number, number of items 
correct on each page, number of items incorrect on each page, number of items 
incomplete or omitted on each page, and the start and stop times for each 
page. 

The computer stored a master list of the total number of items on each . 
page for each of the 37 books in the instructional series. If there was a , 
variance between the number of items on file and the number of items on the 
computer card (as secured from the data log), the computer rejected that 
card. We then went back either to the data log, the computer card, or to 
the actual book which the student had worked on. Initially about five per- 
cent of the datja cards were rejected, due mainly to errors in recording from 
the books themselves. 

The following data report forms were developed: 

A. Status Report (See Appendix B) 

This report lists, by page, all students who have completed a minimum 
number of pages in each book, according- to whether speed and/or accuracy data 
is available for that page and for that student. The status report lists the 
identification number of each student who has completed a specified minimum 
number of pages in the book, and designates with a "T" those pages for which 
we have both speed and accuracy data for the particular student. The desig- 
nation "S" indicates that we only have accuracy data for that page for a 
particular student. The absence of either designation indicates that we have 
neither speed nor accuracy data for that page for the particular student. 

The computer can be instructed to print the status report on a 
calendar basis, e.g., every day, week, month, semester, or print only when a 
minimum number of students have completed a minimum number of pages. This • 
provides us with a means of determining when it is necessary and economical 
to print Intensive reports of individual student performance or reports which 
present data averaged across students. 

B. Book Completion Report by Student (See Appendix C) 

This report lists the number of correct, incorrect, and incomplete 
problems, by page in a particular book, for individual students. We also 
receive, again by page, a measure of performance quality (the percentage 



correct of all problems completed, excluding incomplete problems from the 
computation), as well as performance efficiency (the percentage correct of 
all problems possible on the page, regardless of whether or not they were 
completed by the student). The book completion report by student presents 
elapsed time per page and correct and error rates per minute per page. 

Correct and error rates per minute are computed by dividing the total number 
of correct responses per page by the total time to complete the page (in the 
case of correct rate), and by dividing the total number of incorrect responses 
per page by the total time to complete the page (in the case of error rate). 
The computer also stars (a*sv*a) the five highest correct rates and the five 
highest error rates in each book for the- individual student. This facilitates 
location of those pages on which students experienced little or a great deal 
of difficulty. The- report presents the student's quart! le rank on each page 
for elapsed time and correct rate. 

We also receive the average quality, efficiency, time per page, 
quart! le rank, and correct and error rates across the entire book by student. 
Two averages are involved here; one is computed only for the pages on which 
time was recorded, the other for all pages on which we have any data. 

Finally, the book completion report by student presents cumu 1 at i ve totals for 
all measures, with a separate total computed for pages on which time v;as 
recorded and for pages on which time was not recorded. 

C. Book Comp 1 e t i on Report Across Students, Type 1 (See Appendix D) 

This report contains by class, the following information: book 

number, page number, number of students completing that page, average number 
of items correct, average number of items incorrect, average number of items 
incomplete, average quality and efficiency, total number of items on that 
page, the number of students for which we have speed data on that page, 
average elapsed time, correct and error rates,, as well as the standard error 
of the mean' for elapsed time and correct and error rates. 

L). Book Completion Report Across Students, Type 2 (See Appendix E) 

This report contains by class, the following information: book 

number, page number, number of students completing each page, and the mini- 
mum, first quartile, median, third quartile, and maximum scores for both 
elapsed time per page and correct rate per page. The report can easily be 
expanded to include the minimum, first quartile, median, third quartile, and 
maximum error rates. Finally, this report presents averages for all mea- 
sures (e.g., average median elapsed time per page, average third quartile 
correct rate per page). , 
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RESULTS 



This section is divided into three parts: (A) Pupil Performance Profile, 

(B) The Predictive Model, (C) Comparisons Between Normal and Learning Disabled 
Students. 



A. Pupil Performance Profile 

The major objective of this project was the Initial development of a 
means for describing and evaluating student performance on instructional 
materials under various behavior management procedures. This information 
would be used by teachers, administrators, and others, to determine the effec- 
tiveness of the materials and management procedures, and to predict when 
individual students and types of students will experience difficulty on up- 
coming materials and with particular management procedures. This will enable 
specification of those materials and procedures which have the highest prob- 
ability of proving effective. 

The main measures are as follows: (a) average right, wrong, and 

omitted responses per page for all books on which students worked, (b) average 
quality and standard error per page for all books, (c) average efficiency and 
standard error per page for all books, (d) average elapsed time and standard 
error per page for all books, (e) average correct rate and standard error per 
page for all books, (f) average error rate and standard error per page for all 
books, (g) the number of students for whom we have measures (a) through (c) 
on each page for all books, (h) the number of students for whom we have mea- 
sures (d) through (f) for all books, (i) the total number of responses pos- 
sible per page, (J) the minimum, first quart! le, median, third quartile, and 
maximum elapsed time per page for all books, (k) the minimum, first quartile, 
median, third quartile, and maximum correct rate per page for all books, 

(1) the averages, across all pages within each book for measures (j) and (k) 
for all books, (m) the number of students for whom we have measures (j) and 
(k) on each page for all books. 

Table 1 presents averages, for elapsed tjme and correct rate, of 
the minimum, first quartile, median, third quartile, and maximum scores on 
each of these two measures, by class. 

Table 1 illustrates the superiority of class two (normal students) 
over class one (learning disabled students) on the same books. For example, 
on books 13"I7» .the mean-median elapsed time for class two is lower for every 
book than is the same measure for class one. For the same books, class two 
has a higher mean-median correct rate than class one in four of five instances. 

Figures 1, 2, 3, and ^ present, by class, the minimum, first quar- 
tile, median, third quartile, and maximum score distributions for elapsed 
time and correct rate, by book. 




1 1 

"14 



Ave. 
Book No. 



Mean 

Mean 1st 



ELAPSED TIME 



Mean 



CLASS ONE 
Mean 

3rd Mean 



CORRECT RATE 



Mean 
Mean 1st 



Mean 



Mean 

3rd Mean 



2 


2.1 


78.4 


78.4 


103.4 


131 .6 


134.8 


3.58 


3.58 


4.55 


6.41 


6.84 


3 


3.1 


57.8 


60.2 


99.6 


190.5 


199.4 


2.25 


2.25 


4.32 


7.21 


7.49 


A . 


2.7 


59.6 


■- — — — 

60.2 


97.3 


166.7 


168.7 


2.23 


2.27 


4.38 


7.18 


7.28 


5 


2.3 


49.3 


^9.3 


76..> 


126.2 


126.9 


3.41 


3:41 


5.22 


7.43 


7.63 


13 


5.6 


41 .3 


54.5 




133.7 


165.6 


3.44 


4.23 


6.42 


11.44 


15.67 




6.0 


79.7 


99.8 


144..9 


227.7 


303.8 


2.59 


3.33 


5.06 


8.00 


10.77 


15 


6.2 


53.9 


74.0 


116.9 


212.3 


201.2 


2.35 


3.10 


5.37 


9.05 


13.64 




4.6 


74.4 


05.5 


137.8 


227.6 


259.7 


2.26 


2.67 


4.55 


8.34 


5.91 


17 


4.4 > 


49.9 


56.7 


92.8 


180.2 


215.0 


2.55 


3.16 


6.55 


11.81 


13.35 


18 


2.6 


111 .0 


114.8 


167.3 


237.6 


244.4 


3.;50 


3.73 


6.50 


11.57 


12.80 


19 


1 .1 


107.1 


107.1 


110.7 


113.4 


1 1^-3 


5.83 


5.83 


6.05 


6.18 


6.28 


20 


1 •? 


155.6 


155.6 


209.9 


249.9 


270.9 


3.44 


3.44 


5.29 


7.51 


7.57 


21 


1.8 


212.2 


212.2 


271.2 


297.1 


330.1 


2.72 


2.72 


3.70 


6.12 


6.67 


22 


1 .4 


109.8 


109.8 


129.1 


145.7 


148.4 


6.97 


6.97 

1 


7.87 


8.97 


9.07 
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CLASS TWO 



Ave. 
Book Ho . 



ELAPSED TIME 



Mean 

Mean 1st 



Mean 



Mean 

3rd 



Mean 



CORRECT RATE 
Mean Mean 

Mean 1st Mean 3rd Mean 
Min. Qua rt i 1e Median Quart! 1e Max. 



5 


1 .8 


60.7 


60.7 


77.5 


107.7 


110.7 


3.91 


3.91 


5.39 


6.51 


6.87 


6 


2.9 


50.2 


51.9 


85.0 


131 .0 


132.9 


3.12 


3.19 


A. 92 


9.26 


9.57 


7 


5.2 


A2.1 


50. A 


77. A 


122.9 


IA9.A 


2.25 


2.70 


A.A1 


8.2A 


10.91 


8 


5.7 


26.3 


39.3 


73.5 


120.8 


153.1 


2.21 


2.89 


A. 70 


10.65 


16.02 


3 


6.3 


23.9 


39.6 


77.1 


128.3 


172. A 


1 .83 


2.65 


A. 92 


1 1 .60 


20. A7 


10 


11 .6 


27.1 


53.1 


92.2 


I3A.6 


216.9 


I.5A 


2.73 


A. 25 


8.16 


16. 5A 


11 


8.0 


30.1 


59.6 


88.7 


12A.3 


171.9 


1 .90 


2.88 


A. 33 


6.72 


11 .62 


12 


7.0 


31. A 


A7.6 


71.8 


105.3 


137.1 


3.98 


A. 97 


7.16 


10.95 


18. 72 


13 


12.0 


29.9 


56.1 


85.2 


127.8 


227.7 


2.AA 


A. 07 


6.09 


9.38 


20.25 


lA 


9.9 


60.5 


95.6 


lAO.O 


196. A 


305.9 


2.51 


3.78 


5.17 


7.711 


13.13 


15 


9.2 


39.7 


59.9 


87. A 


123.5 


188.8 


3.33 


A. 72 


6.73 


9.97 


17.26 


16 


5.2 


AO. 2 


56.1 


95.6 


I3A.A 


160. 7 


A. 35 


5.00 


7.65 


13.59 


16.77 


17 


1.9 


3A.A 


3A.A 


A7.7 


69.0 


71.0 


8.57 


8.57 


13.62 


15.79 


18.68 
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CLASS TWO 





Figures 1 - k show that, for class one, there is a definite trend 
toward increases in both elapsed time and correct rate as they proceed 
through the books, particularly for students In the third and fourth quar* 
tiles. For class two, this trend is less definite. For class one, the 
highest (median) elapsed times occur on books 1^, 15, 16, and 18, with the 
lowest (median) elapsed times occuring on books 5, 13, and 17- Also for 
class one, the highest (median) correct rates occur on books 13, 15, 17, 
and l8, with the lowest (median) correct rates occuring on books 2, 3, 
and 16. For class two, the highest (median) elapsed times occur on books 
10, 11, 1^1, 15, and 16, with the lowest (median) elapsed times occuring on 
books 5, 7, 8, 9, and 12. Also for class two, the highest (median) correct 
rates occur on books 12, 13, 15, and 16, with the lowest (median) correct 
rates occuring on books 7, S, 10, and 11. 

Table 2 presents mean quart! le rankings and change in rankings by 
student and by book, for elapsed time and correct rate. The students were 
selected at random, with the only criterion being that they must have com~ 
pleted at least three books. 

The average change in quartile rankings for the learning disabled 
students (class one) was 0.73 for elapsed time and 0.57 for correct rate. 

This represents a change of 1^.6S for elapsed time and 11.1*^ for correct 
rate. The average change in quart! le rankings for the normal students 
(class two) was 0.25 for elapsed time and 0.57 for correct rate. This rep- 
resents a change of 5.0!^ for elapsed time and k .6% for correct rate. For 
both classes, it may be seen that there was more change, from book to book 
for selected students. In regard to elapsed time (1^.6% and 5.0? respectively 
for the two classes), than in regard to correct rate (11.^? and ^.6? respec- 
tively for the two classes). 

The normal students showed considerably less change, on a book to 
book basis for selected students, than did the learning disabled students, 
with the average change (combining elapsed time and correct rate) being k.8% 
for the former and 13.0? for the latter. Normal students showed a change of 
more than one full ranking on the average of one out of 23 opportunities, 
while for learning disabled students this occured on the average of one out 
of four opportunities. 

Table 3 presents mean elapsed time and correct rate, and change in 
these measures, by class and student. The data presented are for the same 
students as those in the preceding table. 



The average change in absolute (not quartlle rankings) elapsed t^mes 
and correct rates for the learning disabled students (class one) was 57*1'^ 
and ^6.0? respectively for the two measures. The average change in absolute 
scores for the same measures for normal students (class two) was 35.1^ and 
21.1? respectively. 

As was the case with quartile rankings, when assessing changes in 
absolute elapsed time and correct rate measures from book to boo!;, <or both 
classes of students, there was more change in regard to elapsed time (57«1'^ 
and 35.1?) than for correct rate (^6.0? and 21.1?). 
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CLASS CME 





Subj . 
No. 


Book 

No. 


Mean Quart lie 
Rank : 

Elapsed Time 


Change from 
Last Book 


Mean Quart! le 
Rank : 

Correct Rate 


•Change from 
: Last Book 


205 




2.70 


m 


2.60 . 


- 


205 


15 


4.17 


1.47 


3.96 


1.36 .. 


205 


16 


2.37 


1.80 


2.49 


1.47 ■■ 


205 


17 


3.36 


0.99 


3.07 


0.58 


205 


IS 


2.79 


0.57 


3.23 


0.16 


207 


15 


1.99 


- 


1.96 


- 


207 


16 


1.34 


0,65 


1 .63 


0.33 


207 


17 


1.86 


0.52. 


1.75 


.0.12 


303 


13 


2.9.5 




2.76 




303 


15 


2.91 


0.04 


2.69 


0.07 


303 


16 


3.25 


• . 0.34 


3.25 


0.56 


307 


13 


1.68 




1.66 




307 


\k 


1.84 


0.16 


1.74 


0.08 


307 


15 


3.04 


1.20 - 


2.93 


1.19 


308 


13 


2.62 


- 


2.57 


■ - 


308 


ll» ‘ 


3.27 


o.«65 


3.22 


0.65 


308 


15 


2.96 


0.31 


2.93 


0.29 



Average: 0.73 0»57 



TABLE 2, PAGE 1 : MEAfl aUARjTlLE RANKINGS AND RANKING CHANGES, BY CLASS 
AND ST'J-JENT FOu" ELAPSED' T H;E AilC CORRECT' RATE 
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CLASS TWO 



Subj . 
No. 


Book 

No. 


Mean Quart! le 
Rank: 

Elapsed Time 


Change from 
Last Book 


Mean Quart lie 
Rank : 

Correct Rate 


Change from 
Last Book 


^01 


14 


2.25 


- 


2.12 


. - 


^♦01 


15 


2.17 


0.08 


2.16 


0.04 


^01 


16 


2.55 


0.38 


2.37 


0.21 


^♦07 


13 


3.97 


- 


3.84 


- 


k07 


14 


4.18 


0.21 


4.11 


0.27 




15 


4.11 


0.07 


3.99 


0.12 


l\07 


16 


3.88 


0.23 


3.73 


0.26 


^♦05 


13 


2.52 


- 


2.40 


- 


ko^j 


14 


2.70 


0.18 


2.48 


0.08 


kQ3 


15 


1 .82 


0 .88 


1.69 


0.79 


415 


13 


2.32 


- • 


2.35 


- 


.415 


14 


3.65 


1.33 


3.58 


1.23 


415 


15 


3.34 


0.31 


3.15 


0.43 


416 


13 


2.01 


- 


1.91 


- 


416 


14 


2.32 


0.31 


2.05 


0.14 


416 


15 


2.00 


0.32 


1.89 


0.16 


418 


13 


1.71 


- 


1.59 


- 


418 


, 14 


1.78 


0.07 


1.78 


0.19 


418 


15 


2.29 


0.51 


2.23 


0.45 


420 


13 


2.92' 


- 


2.88 


- 


420 


14 


3.14 


0.22 


2.93 


0.05 


420 


15 


3.49 


0.35 


3.33 


0.40 


420 


16 


4.14 


0.65 


3.86 


0.53 


422 


13 


2.73 




2.65 


■ - 


422 


14 


2.41 


0.32 


2.35 


0.30 


422 


15 


2.97 


0.56 


2.91 


0.56 


423 


13 


1.70 


- 


1.75 


- 


423 


14 


1 .90 


0.20 


1.96 


0.21 


423 


15 


1.42 


0.48 


1.62 


0.34 


423 


16 


1.13 


0.29 


1.46 


0.16 


425 


13 


1.77 


- 


1 .82 


- 


425 


14 


2.05 


0.28 


2.05 


0.23 


425 


15 


2.21 


0.16 


2.21 


0.16 



Average: 0.25 ($.0%) 0.23 (^.6^) 

TABLE 2, PAGE 2 : MEAN QUART I LE RANKINGS AND RANKING CHANGES, BY CLASS 

AND STULL..-T FUk ELAPSED TIME A!iU CORRECT RATE 
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CLASS ONE 




Subj. Book Mean Change from Mean Change from 

No . No. Elapsed Time Last Book(^) Correct Rate Last Book(^ ) 



205 




159 




5.07 


— 


205 


15 


66 


58.5 


ll.Olt 


217.7 


205 


16 


162 


2it5.5 


5.lt5 


50 .7 


207 


15 


163 




5.00 


— 


207 


16 


217 


33.1 


3-3lt 


33.2 


207 


17 


. Iit5 .. 


33.2 


5.19 


55 . 1 * 


303 


13 


107 




7.05 


f 


303 


15 


131 * 


25.2 


7.it3 


5.1* 


303 


16 


119 


11.2 


. . 8.01 


7.8 


307 


13 


129 . 


-- 


5.lt6 


' -- 


307 




20'lt' 


58.1 ; 


it.2lt 


22.3 


. 307 


15 


lOlt 


^ 9.0 


6.lt7 


52.6 


308 


’3 


96 


■ — 


6.6lt 


' ' 


308 


lit 


130 


lilt.lt 


6.37 


4.2 


308 


15 


113 


13.1 


7.02 


10.2 



Average: 57.1^ k6.0% 



MEAN ELAPSED TIME AMD CORRECT RATE, 
IN .THESE, BY CLASS AND STUDENT 



TABLE 3, PAGE 1 ; 
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1 



CLASS TWO 



Subj . 
No. 


Book 

No. 


Mean 

Elapsed Time 


Change from 
Last Book 


Mean 

Correct Rate 


Change from 
Last Book 


^♦01 


14 


178 


r 


5.08 


-- 


^♦01 


15 


104 


41 .6 


7.88 


55.1 


^♦01 


16 


92 


11.6 


8.19 


■ 3.8 


^♦07 


13 


50 


-- 


12.53 


-- 


^♦07 


14 


82 


64.0 


10.08 


19.6 


^♦07 


15 


53. 


35.4 


12.86 


27.5 




16 


52 


1.9 


13.99 


8.7 




13 


93 




6.14 


— 


^♦oy 


14 


142 


43.4 


5.34 


13.2 


^oy 


15 


116 


18.3 


5.40 


1.1 


^15 


13 


99 


-- 


6.4o’ 


— 


^15 


14 


100 


1 .0 


8.25 


28.9 


^15 


15 


72 


28.0 


9.81 


18.9 


^♦16 


13 


115 




5.48 


\ 


^16 


14 


152 


32.7 


4.87 


11.1 


^16 


15 


117 


23.1 


5.62 


15.6 




13 


126 


-- 


4.53 




^♦18 


14 . 


185 


46.3 


4.47 


i.4 


4l8 


15 


102 


44.7 


6.50 


45.4 


^♦20 


13 


77 




7.45 


— 


^♦20 


14 


115 


49.2 


6.90 


7.4 


^♦20 


15 


67 


41.7 


9.69 


40.4 


420 


16 


49 


26.8 


14.69 


51.6 


422 


13 


83 


-- 


7.30 


— 


422 


14 


158 


90.4 


5.44 . 


24.3 


422 


15 


77 


51.3 


8.42 


54.8 


423 


13 


129 


-- 


5.10 


-- 


423 


14 


178 


37.9 


4.79 


6.1 


423 


15 


126 


29.2 


5.39 


12.5 


423 


16 


130 


3.2 


5.32 


1.3 


425 


13 


129 


— 


5.17 


— 


425 


14 


195 


51.2 


4.90 


5.3 


425 


15 


•130 


33.4 


6.49 


32.5 


Average : 




35.1^ 




21. n 



TABLE 3, PAGE 2 ; MEAN ELAPSED TIME AND CORRECT RATE, 
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Again, the normal students showed considerably less change, on a 
book to book basis for selected students, than learning disabled students, 
with the average change (combining elapsed time and correct rate) being 
28.1% for the former and 51*8% for the latter. Normal students showed a 
change of more than i0% in either of the two measures on 19 out of oppor- 
tunities (or i»l .3% of these opportunities), while learning disabled students 
showed a cliange of more than 30% on 12 out of 20 opportunities (or 60..0% of 
these opportunities). 

Comparison of Table 2 with Table 3 shows that absolute scores for 
elapsed time and correct rate showed much more change, from book to book for 
selected students, than did quartile ranking changes. The average percent- 
age change for these two measures was 3.57 times as large for absolute scores 
as for QMartile rankings, in the case of learning disabled students. For 
normal students , the average percentage change for these two measures was 
5.85 times as large for absolute scores as for quart! le rankings. 

Table k presents changes in quartile rankings for correct rate and 
elapsed time within a given book (in this case book 13) for selected students 
from class two. The purpose of this table is to show that there is little 
change in quartile rankings, for individual students, in relation to per- 
formance throughout a given book. This corresponds to the finding of 
stability In quartile rankings across books, particularly for the normal 
students. 



It will be noted that the data for the eight students presented in 
Table A shows considerable stability, particularly for Students AlO, Al5, 

Al6, and Al8. For example. Student 4l8 has mean quartile rankings, for 
correct rate, of 3«3» 3.0, and 3.0 respectively, for the first 50, the second 
50, and the last 28 pages in book 13- The same student has mean quartile 
rankings, for, elapsed time, of 3.0, 2.9» and 2.9 respectively, for the first 
50, the second 50, and the last 28 pages in the same book.' Only Students ^20 
and ^22 show any appreciable changes and these are not major. For example. 
Student h20 shows a change of 0.5 quartile rankings from the first 50 to the 
second 50 pages in book 13> snd a change of 0.6 quartile rankings from the 
second 50 to the last 28 pages in the same book, in regard to correct rate. 

In regard to elapsed time, the same student shows a change of O.A quartile 
rankings from the first 50 to the second 50 pages in book 13, and a change 
of 0.6 quartile rankings from the second 50 to the last 28 pages in the same 
book. 



B. The Predictive Model 

As mentioned in the Objectives section, there are many variables 
which can be used in an equation or series of equations designed to predict 
performance on instructional materials. In the present project a very simple 
predictive equation was utilized, one which contained only a few variables. 

In order to construct the equation one need know only the following infor- 
mation (focusing here on correct rate): 

1. The average correct rate of the individual student's reference 
group (the reference group is composed of students whose char- 
acteristics are similar to those for the student whose per- 
formance we are trying to predict. These characteristics may 



(Book 13) 


Mean QR-CR 
1st 50pp 


Mean QR-CR 
2nd.50pp 


Mean QR-CR 
last 28pp 


Mean QR-ET 
i 1st 50pp 


. Mean QR-ET 
2nd 50pp 


Mean QR-ET 
last 28pp 


407 


0.7 


O.b 


0.4 


0.7 


0.5 


0.4 


410 


3 . ‘3 


4.3 


3.9 


4.2 


4.5 


4.1 


415 . 


2.2 


2.3 




2.1 


2.2 


- 


416 


. , 2.7 


2.8 


3.2 


2.6 


r 

2.6 


3.0 


418 


3.3 


3.0 


, 3.0 


3.0 


2.9 


2.9 


420 


1.2 


1.7 


2.3 


1.2 


, 1.6 


2.2 


422 


2.0 


2 . 0 . 


1.5 


1.9 


2.1 


1.4 


423 


3.0 


3.4 


i ' 2.6 


2.9 


i . 


2.7 



TABLE 4 ;, .CHANGES IN QUART I LE RANKINGS WITHIN A GIVEN BOOK 



be the students' age, sex, performance on related materials, 
performance. on standardized tests, other measures, or a com- 
bination of measures) on the last book which the Individual 
student has completed^ 

2. the correct rate of the individual student's reference group 
on upcoming pages or books (i.e., the pages or books on which 
we are trying to predict the individual's performance). 

3. the differences between 1. and 2., noting the direction of 
the difference (i.e., was there an increase or a decrease from 
I. to 2,?). For purposes of the predictive equation, this 
difference Is divided by five (5) » since this corrected fairly 
well for the fact that, in some Instances, when a few students 
in the reference group had extreme scores, this led to con- 
siderable skewing of the averaged measures for the group. 

the correct rate of the individual student on the last book 
he or she completed. 

5. the Interquartile rank of the Individual student on the last 
book he or she completed^ 

Given the preceding Information, in addition to the basic assumption 
that the individual student will have the same interquart i le rank on the up- 
coming book as he of she had on the previous book in relation to the perform- 
ance measured, we are ready to proceed to the implementation of the predictive 
equation. 

Assume that the student whose performance we are trying to predict 
had a quart! le ranking on his or her last book of 2.7^. We will then assume 
that this student's quart! le ranking on the upcoming book wMl also be 2.7^. 
The next step is to secure the Book Completion Report Across Students, Type 2 
(see Appendix E) for the book in which we are going to predict the student's 
performance. This report lists, for both elapsed time and correct rate 
(error rate may also be listed, but was not during the current project), the 
average minimum, first quart! le, median, third quartile, and maximum scores 
for the entire book, as well as for each of the pages in the book. 

Assuming a predicted quartile rank for our student of Interest of 
2.7^, we determine what correct rate is represented by quartile ranking of 
1.1k. This calculation is made possible by the fact that we have print-outs 
of the average (for the book) first quartile, median, third quartile 
scores, etc. Interpolation Is required when the student's anticipated 
quartile rank is not exactly 1.0, 2.0, e.O, or 4.0. 

Tables i> through 8 present predicted elapsed times and correct 
rates for students selected at random from the two classes. 
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Mean-' Dlff; Bet.‘ . Actual' Actual Predicted Predicted Last Book Predic- 
Book Class’ Median This 6 Next ’Mean:’ Mean Mean Mean off by tion off 
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table 5 :- PREDICTED ELAPSED TIME AND RkLATED DATA, CLASS TWO 
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TABLE 6; PREDICTED CORRECT RATE AND RELATED DATA, CLASS TWO 



30 



27 



i 



o 

ERIC 



DIff. Bet. Actual Actual Predicted Predicted Last Book Predic- 



Book Class Median This & Next 



if a 



Mean Mean Mean Mean 



ET 



book-ET sa ET QR-ET QR-ET 



ET 



off-,by tion off 
{%) by {%) 





1 


\ kk .3 


(-28.0) 


205 


159.0 


2.70 


— 


— 


— 


— 


15 


1 


116. <) 


(+30.9) 


205 


.66.0 


^•17 


2.70 


98. 


li»0.9 


i»9.1 


16 


1 


137.8 


( - ) 


205 


162.0 


2.37 


,C17 


86.2 


59.2 


l»6.7 


l3 




,85. A 


(+59.5) 


o 


129.0 


1.68 


• ■ 


— 


— 


— 


1^» 


1 


\ kk .3 


(-28.0) 


307 


20l».0 


1.8i» 


1,68 


207.1 


36.7. 


1.5 


15 


1 


116.9 


v( - ) 


307 


lOlj.O 


3.0i» 


1;;81» 


191 •‘i 


96.1 


8l».0 


13 


1 


85 .i» 


(+59.5) 


308 


90.0 


2.62 




— 


— 




1^ 






(-23.0) 


308 


130.0 


3.27 


2.62 


139.7 


30.8 


7-5 


15 


1 


116.9 


( - ) 


308 


113.0 


2.96 


3.27 


63.0 


15.0 


kk.Z 



Ave«= 

S3.U 



TABLE 7: PREDICTED ELAPSED tImE AND RELATED DATA, CLASS ONE 
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TABLE 8;* PREDICTED CORRECT RATE AND RELATED DATA, CLASS ONE 



Table 5 shows that, using the formula just described, our predic- 
tions were, on the^average, within less than 1^1^ of the actual elapsed times 
for class two. If we use the students' average elapsed time on the previous 
book to predict their performance on the next book our estimates are off, on 
the average, by 33. 9% for class two. It should be noted that there is a 
constant underestimation of elapsed time on book 15 ^for the three selected 
students. For these students, our predictions, using the formula, are off 
by 25.9%, 30.5%, and . 36.1%, respectively. If this error w® re found to be 
maintained across many students on this book, a constant factor could be 
added to the predicted elapsed time, which would bring tiie average error in 
prediction to approximately five to eight percent. 

Table 6 shows that, using the formula, our p red let ions; 'We •‘e* on the 
average, within less than 12% of the actual correct rates for class two.. If 
we use the students' average correct rates on the previous book to predict 
their performance on the next book our estimates are off, on-the average, by 
15.^%. It should be noted that one student contributed very heavily to the 
average deviation from predicted scores. 

For class one, the learning disabled students, our predictive equa- 
• tion was less accurate than for the normal students. Table 7 shou/s that the 
prediction for elapsed time vyas off, on the average, 38.8% (compared to 13.9^ 
for the normal class). Using the students' elapsed time on the previous book 
■ as a predictor of performance on the upcomi ng book produced an average error 
of 63.1%.. Table 8 shows that the prediction for correct rate of the learning 
disabled students v;as off, on the average, 22.9% (compared to 11.^% for the 
•normal class). Using the students' correct rate on the previous book pro- 
duced an average error of prediction of 38.9%. 

For both the learni ng disabled and the normal students , predi ctions 
of correct rate were more accurate than predictions for elapsed, time. This 
difference was especi al ly. pronounced for the learning disabled students. 



C. Comparisons Between iiormal and Learning Disabled Students 

As mentioned in part A. of this section and presented in Table 1, 
the normal students performed better than the .learning disabled students on 
an average of five of every six measures for those books for which data was 
available for both classes. 

Figure 5 depicts mean-median elapsed times for all books for which 
data is available for both classes. 

Examination of Figure 5 shows that the differences in mean“medlan 
elapsed time between class one and two increase progressively after book 1^, 
with class two being superior to class one in each book except book 5. 

Figure 6 depicts the mean-median correct rates for all books for 
which data is available for both classes. 

Figure 6 sha-/s very clearly the progressive Increase in differences 
in mean-median correct rate after book 1^, analogous to that shown in regard 
to elapsed time in Figure 5. Again the normal students are superior to the 
learning disabled students, except on book 13. 
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DISCUSSION AND CONCLUSIONS 



The Pupil Performance Profile, to date, has had little more than a 
preliminary development. The results so far are only a gross approximation 
of the conciseness and preci s ion whl ch eventually should be possible utiliz- 
ing this approach. However, even relying solely on the capabilities of the 
present system,, considerable, information may be made available for decision- 
making by teachers, administrators, parents, and students. 

The present capabilities of the system are as follows: 

(a) ' the determi nat ion. of student performance on three major measures 

by class or type, of student; 

(b) the determination of performance on each measure by individual 
■ student; 

(c) the relationship of individual students' performances to that of 
their reference group (i.e., those in the same class and/or of 
the same type; e.g., learning disabled, normal, brain-damaged, 
physically handicapped); 

(d) the distribution of student performance by quartile ranking; this 
enables the. determi nat ion of the range of scores which encompass 
25^, 50^, or 75% .of all students on each page in every book and 
for each book as a whole; 

(e) the prediction of student performance, using the predictive 
formula discussed in the Results section. 

For the purposes of this pilot project, fine-grain data was gathered 
and printed out for every student on every page on which he or she worked. 

In actual classroom appl icaj^ion, however, whi le continuous measurement and 
• recording would be maintained, the type, frequency, and amount of print- 
outs could be reduced sharply wi thout loss of effectiveness and with con- 
siderable gain in efficiency. For example, instead of presenting teachers 
v/lth complete print-outs, by page and by book for each student (see Appendix 
C) , a print-out could be presented which indicated only the student's 
average data for the measures of interest. Then, if the data so indicated, 
a complete print-out could be requested. This capsule summary would allow 
a teacher or administrator to scan but one or two pages in order to deter- 
mine how all students are performing. The same condensation tactic may be 
used in regard to other reports, such as student performance by class 
quartile rankings, (see Appendix E) . 

In similar fashion, separate print-outs of predicted performance, by 
page and/or by book, could be made available so' that decision-makers could 
•be alerted to the likelihood of deficient performance and be provided with 
sufficient lead time to deal with these situations. 
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Overall, the relative performance of the two classes of students was 
as expected. The superiority of the normal students, despite the fact that 
their mean agei'was several years less than the learning disabled students, 
came as no surprise. Some of the criteria for classifying students as 
learning disabled are inattentiveness, lack of concentration, and the exist- 
ence of competing behaviors; e.g., out-of-seat, talk-outs, which are incom- 
patible with high-level academic performance. The learning disabled students 
are not, on either an inter- or an intra-student basis, uniformly poor 
performers. Rather, their performance is erratic, ranging from scores which 
equal or exceed the normal students on certain pages and for certain books, 
to scores which are abysmally low. 

indeed, for both classes, there was an inverse relationship between 
variability in performance within and across books and overall level of 
performance. That is, the best students shaved the most consistency, the 
better (but not best) students were less consistent than the best students, 
but more consistent than the poorer students, and so on. The worst students 
were the least consistent. The implication of this finding is that predic- 
tions of the performance of poorer students will, on the average, be less 
accurate than predictions of the performance of better students. 

Offsetting the previous observation is the fact that, in the current 
study, despite considerable fluctuations In scores of individual students 
from one book to the next, students tended to fluctuate at the same points. 
Take the case of a student who had a correct rate on book 13 of 0.0 per 
minute, followed by correct rates of 17*0 and 6.8 on books 1^ and 15, 
respectively. This degree of variability from book to book would make 
prediction quite difficult, were it not for the fact that, lumping all 
students of a given class or type, their averaged scores across these same 
books changed in the same direction as that of the individual student. 

The individual student's quartile rank, by page and averaged across 
an entire book, because of its demonstrated stability (e.g., average 
changes for normal students of approximately five percent from book to 
book for selected students), makes for accurate prediction. This is the 
case despite the very elementary nature of the predictive formula. In the 
future, the formula could be expanded to include the fol lotting: 

(a) past performance across more than one book and trends in past 
performance; 

(b) the history of the accuracy of past predictions; 

(c) the Inclusion of other variables in the formula; 

(d) the determination of appropriate weighting factors for the 
variables used; 

(e) the development of sub-formulas which are accurate predictors 
of the performance of individuals or . small groups of students, 
rather than using a universal formula applied to all students; 



(f) provision for predictions which cover a range of values, 
e.g., a correct rate of 6.2 to 7*5; 

(g) provision for stating the probability of correctly predicting 
a score; e.g., predicting, with a probabi 1 ity of ..OJi, an 
average correct rate on book 22. of 13.1, with a range from 
7.8 to 10.3. 

Difficulties Encountered in Carrying Out Research 

The major difficulty encountered in carrying out the research stemmed 
from the issuance of a directive wiilch precluded the use of students at the 
Experimental Education Unit, CDHRG. The denial of these students forced a 
change in locale to the Mercer View School Mercer Island, Washington. In 
the time availablej it was not- possible to secure the range of student char- 
acteristics, the precise matching of students, nor the degree of experimental 
control which wouid'have pertained had the project been carried out at the 
Experimental Education Unit. 

The original experimental design Included the use of different treatment 
conditions with students of similar characteristics, and the application of 
the same treatment to students of different characteristics. This design 
would have provided precise comparative data which would have enabled us to 
make at least a gross determination of the degree of contribution of several 
factors, including the student, the instructional materials, and the behavior 
management procedures , to student performance. Instead, we were limited, 
primarily by lack of time, but also by manpower (since working in Mercer 
Island required far greater expenditures for travel and consumed much time), 
to the use of one class of normal students (second grade) and one class of/ 
learning disabled students (ages six to eleven). As a result, it was not'’ 
possible to make detailed comparative assessments of the contributions to 
performance of the aforementioned three elements. 

However, the principal investigator still strongly feels that such an 
approach is feasible and will yield worthwhi le results. 

Probably the most important problem was the failure of our initial 
computer programmer to fulfill his obligations. As a result, a delay of 
several months was encountered until his duties were taken over by someone 
else. Vihereas the computer was supposed to have been programmed to handle 
predictions of student performance, it became necessary, as a result of 
this delay, to work the predictions by hand. Thus we had considerably less 
opportunity to test the formula and almost no chance to make it more sophis- 
ticated and effective. 

The other significant problem v;as an inherent part of the project it- 
self, and, as such, could not have been avoided. This refers to the extreme 
difficulty of determining, in advance, the cost of carrying out various 
mathematical operations by computer. As the project progressed, we were 
far better able to predict what our costs would be and hence were able to 
stay within the budget. 



SUMMARY AMD RECOMMENDATIONS 



This project accomplished most of wliat it proposed to do, with the 
exception of providing comparative data concerning the relative contribu- 
tions to performance of student characteristics, instructional materials, 
and behavior management procedures (the reason for not accomplishing the 
latter objective Is presented in the previous section). The project has 
resulted in the initial development of an objective system for evaluating 
and describing student performance, as well as for predicting performance. 

The cost of Installing and operating such a system on a mass basis, 
once It becomes fully operational, should be low. Any school or school 
district which has teletype access to a computer will be able to partici- 
pate in this activity. Perhaps the major hurdle will be the development 
of more efficient procedures for scoring student responses and for trans- 
ferring the responses to punch cards or other media. It Is possible that 
instructional materials may be developed which provide for responses which 
can be read by the computer Itself. EJut even without this degree of 
sophistication. It should be possible to find, at low cost, students or 
aides who would serve as data recorders. 

The current project's principal investigator plans to expand the pro- 
ject to include a wide range of types of students, instructional materials, 
and behavior management procedures. This phase of the research, which will, 
require approximately three years for completion, will be . prel iminary to 
actual implementation on a large scale basis. 
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